Crowdsourcing Supplemental Content

ABSTRACT

Methods and systems for sourcing supplemental content are disclosed. Secondary devices may be used to identify content streaming on first screen devices and to generate supplemental data for the content. In this manner, users may be leveraged to create various data for a variety of content. The data may be collected and organized so that users watching content at a later time may have access to the data. Methods and systems for using second screen devices to access metadata created by the crowd are also disclosed.

BACKGROUND

Television viewing is no longer the static, isolated, passive pastimethat it used to be. Today, viewers have the option of using a computingdevice, such as a tablet computer, to view a webpage related to a showthey are watching, thereby keeping the viewers engaged in a particularprogram. The related content, however, requires significant amounts ofdata related to the show to keep the viewers interested. There remains aneed to efficiently gather and provide related information of interest.

SUMMARY

Some of the various features described herein may facilitate acquiringdata, such as metadata, and associating the metadata with content. Inparticular, some of the systems described below allow users to supplymetadata related to the content they are consuming using their own userdevices.

In accordance with aspects of the disclosure, users, such as subscribersor ordinary consumers (e.g., the “crowd”), may be leveraged to generateand organize metadata content for enhancing consumption of primarycontent. In an illustrative embodiment, the disclosure teaches a methodcomprising streaming content to one or more users. While the content isdelivered to a first screen device (e.g., a television, computermonitor, mobile device, etc.), users may generate submissions usingsecond screen devices (e.g., smartphones, laptops, tablets, etc.). Thesubmissions, e.g., tag submissions, may include data (which may be madeinto metadata) relevant to the content. The data may also includeinformation identifying a time point in the content to which the dataapplies. The tag submissions may be transmitted to another device, whichmay generate metadata tags using the received tag submissions.Subsequently, the metadata tags may be supplied to second screendevices. Additionally, some aspects of the disclosure relate tocomputing devices, having a processor and memory storingcomputer-executable instructions, and other apparatuses to perform theabove steps and other steps for improving a second screen experience.

Other details and features will also be described in the sections thatfollow. This summary is not intended to identify critical or essentialfeatures of the inventions claimed herein, but instead merely summarizescertain features and variations thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

Some features herein are illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements.

FIG. 1 illustrates an example communication network on which variousfeatures described herein may be used.

FIG. 2 illustrates an example computing device and softwareconfiguration that can be used to implement any of the methods, servers,entities, and computing devices described herein.

FIG. 3 illustrates a system architecture according to one or moreillustrative aspects described herein.

FIGS. 4A-4H are diagrams illustrating an example embodiment of aspectsof the present disclosure.

FIG. 5 is a flow diagram illustrating an example method according to oneor more aspects of the disclosure.

FIG. 6 is a flow diagram illustrating an example method according to oneor more aspects of the disclosure.

FIG. 7 is a flow diagram illustrating an example method according to oneor more aspects of the disclosure.

FIG. 8 is a diagram illustrating an example embodiment according to oneor more aspects of the disclosure.

FIG. 9 is a diagram illustrating another example embodiment according toone or more aspects of the disclosure.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

By way of introduction, some features described herein may allow a userto consume content (e.g., audio visual content such as a televisionprogram) on one device (e.g., a television, smartphone, tablet, laptop,etc.) and generate metadata associated with the content using a seconddevice (e.g., a television, smartphone, tablet, laptop, etc.). In oneexample, a smartphone may be adapted to automatically detect/identifythe content that a user is consuming from the audio associated with thatcontent, and may allow a user to submit data to be associated with thecontent. Further, there may be a system that allows many users tocontribute various types of data for the same content. Thus, when otherusers subsequently consume the content (e.g., television programs, musicvideos, live events, home videos, etc.), the other users may access thedata, including any associated information such as webpages, using theirsmartphones or other devices. Accordingly, an aspect of the presentdisclosure is to crowdsource the creation of metadata.

FIG. 1 illustrates an example communication network 100 on which many ofthe various features described herein, such as the requesting andretrieval of content and metadata and/or the delivery of metadata to acentral database, may be implemented. Network 100 may be any type ofinformation distribution network, such as satellite, telephone,cellular, wireless, etc. One example may be an optical fiber network, acoaxial cable network, or a hybrid fiber/coax distribution network. Suchnetworks 100 use a series of interconnected communication links 101(e.g., coaxial cables, optical fibers, wireless, etc.) to connectmultiple premises 102 (e.g., businesses, homes, consumer dwellings,etc.) to a local office or headend 103. The local office 103 maytransmit downstream information signals onto the links 101, and eachpremises 102 may have a receiver used to receive and process thosesignals.

There may be one link 101 originating from the local office 103, and itmay be split a number of times to distribute the signal to variouspremises 102 in the vicinity (which may be many miles) of the localoffice 103. The links 101 may include components not illustrated, suchas splitters, filters, amplifiers, etc. to help convey the signalclearly, but in general each split introduces a bit of signaldegradation. Portions of the links 101 may also be implemented withfiber-optic cable, while other portions may be implemented with coaxialcable, other lines, or wireless communication paths. By running fiberoptic cable along some portions, for example, signal degradation may besignificantly minimized, allowing a single local office 103 to reacheven farther with its network of links 101 than before.

The local office 103 may include an interface, such as a terminationsystem (TS) 104. More specifically, the interface 104 may be a cablemodem termination system (CMTS), which may be a computing deviceconfigured to manage communications between devices on the network oflinks 101 and backend devices such as servers 105-107 (to be discussedfurther below). The interface 104 may be as specified in a standard,such as the Data Over Cable Service Interface Specification (DOCSIS)standard, published by Cable Television Laboratories, Inc. (a.k.a.CableLabs), or it may be a similar or modified device instead. Theinterface 104 may be configured to place data on one or more downstreamfrequencies to be received by modems at the various premises 102, and toreceive upstream communications from those modems on one or moreupstream frequencies.

The local office 103 may also include one or more network interfaces108, which can permit the local office 103 to communicate with variousother external networks 109. These networks 109 may include, forexample, networks of Internet devices, telephone networks, cellulartelephone networks, fiber optic networks, local wireless networks (e.g.,WiMAX), satellite networks, and any other desired network, and thenetwork interface 108 may include the corresponding circuitry needed tocommunicate on the external networks 109, and to other devices on thenetwork such as a cellular telephone network and its corresponding cellphones.

As noted above, the local office 103 may include a variety of servers105-107 that may be configured to perform various functions. Forexample, the local office 103 may include a push notification server105. The push notification server 105 may generate push notifications todeliver data and/or commands to the various premises 102 in the network(or more specifically, to the devices in the premises 102 that areconfigured to detect such notifications). The local office 103 may alsoinclude a content server 106. The content server 106 may be one or morecomputing devices that are configured to provide content to users attheir premises. This content may be, for example, video on demandmovies, television programs, songs, text listings, etc. The contentserver 106 may include software to validate user identities andentitlements, to locate and retrieve requested content, to encrypt thecontent, and to initiate delivery (e.g., streaming) of the content tothe requesting user(s) and/or device(s).

The local office 103 may also include one or more application servers107. An application server 107 may be a computing device configured tooffer any desired service, and may run various languages and operatingsystems (e.g., servlets and JSP pages running on Tomcat/MySQL, OSX, BSD,Ubuntu, Redhat, HTMLS, JavaScript, AJAX and COMET). For example, anapplication server may be responsible for collecting television programlistings information and generating a data download for electronicprogram guide listings. Another application server may be responsiblefor monitoring user viewing habits and collecting that information foruse in selecting advertisements. Yet another application server may beresponsible for formatting and inserting advertisements in a videostream being transmitted to the premises 102. Although shown separately,one of ordinary skill in the art will appreciate that the push server105, content server 106, and application server 107 may be combined.Further, here the push server 105, content server 106, and applicationserver 107 are shown generally, and it will be understood that they mayeach contain memory storing computer executable instructions to cause aprocessor to perform steps described herein and/or memory for storingdata, such as information for identifying a user, content audio filesfor identifying content from an audio profile or audio clip, andmetadata for viewing on second screen devices.

An example premises 102 a, such as a home, may include an interface 120.The interface 120 can include any communication circuitry needed toallow a device to communicate on one or more links 101 with otherdevices in the network. For example, the interface 120 may include amodem 110, which may include transmitters and receivers used tocommunicate on the links 101 and with the local office 103. The modem110 may be, for example, a coaxial cable modem (for coaxial cable lines101), a fiber interface node (for fiber optic lines 101), twisted-pairtelephone modem, cellular telephone transceiver, satellite transceiver,local wi-fi router or access point, or any other desired modem device.Also, although only one modem is shown in FIG. 1, a plurality of modemsoperating in parallel may be implemented within the interface 120.Further, the interface 120 may include a gateway interface device 111.The modem 110 may be connected to, or be a part of, the gatewayinterface device 111. The gateway interface device 111 may be acomputing device that communicates with the modem(s) 110 to allow one ormore other devices in the premises 102 a, to communicate with the localoffice 103 and other devices beyond the local office 103. The gateway111 may be a set-top box (STB), digital video recorder (DVR), computerserver, or any other desired computing device. The gateway 111 may alsoinclude (not shown) local network interfaces to provide communicationsignals to requesting entities/devices in the premises 102 a, such asdisplay devices 112 (e.g., televisions), additional STBs 113, personalcomputers 114, laptop computers 115, wireless devices 116 (e.g.,wireless routers, wireless laptops, notebooks, tablets and netbooks,cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones),mobile phones, mobile televisions, personal digital assistants (PDA),etc.), landline phones 117 (e.g. Voice over Internet Protocol—VoIPphones), and any other desired devices. Examples of the local networkinterfaces include Multimedia Over Coax Alliance (MoCA) interfaces,Ethernet interfaces, universal serial bus (USB) interfaces, wirelessinterfaces (e.g., IEEE 802.11, IEEE 802.16), analog twisted pairinterfaces, Bluetooth interfaces, and others.

FIG. 2 illustrates general hardware elements that can be used toimplement any of the various computing devices discussed herein. Thecomputing device 200 may include one or more processors 201, which mayexecute instructions of a computer program to perform any of thefeatures described herein. The instructions may be stored in any type ofcomputer-readable medium or memory, to configure the operation of theprocessor 201. For example, instructions may be stored in a read-onlymemory (ROM) 202, random access memory (RAM) 203, removable media 204,such as a Universal Serial Bus (USB) drive, compact disk (CD) or digitalversatile disk (DVD), floppy disk drive, or any other desired storagemedium. Instructions may also be stored in an attached (or internal)hard drive 205. The computing device 200 may include one or more outputdevices, such as a display 206 (e.g., an external television), and mayinclude one or more output device controllers 207, such as a videoprocessor. There may also be one or more user input devices 208, such asa remote control, keyboard, mouse, touch screen, microphone, etc. Thecomputing device 200 may also include one or more network interfaces,such as a network input/output (I/O) circuit 209 (e.g., a network card)to communicate with an external network 210. The network input/outputcircuit 209 may be a wired interface, wireless interface, or acombination of the two. In some embodiments, the network input/outputcircuit 209 may include a modem (e.g., a cable modem), and the externalnetwork 210 may include the communication links 101 discussed above, theexternal network 109, an in-home network, a provider's wireless,coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., aDOCSIS network), or any other desired network.

The FIG. 2 example is a hardware configuration. Modifications may bemade to add, remove, combine, divide, etc. components of the computingdevice 200 as desired. Additionally, the components illustrated may beimplemented using basic computing devices and components, and the samecomponents (e.g., processor 201, ROM storage 202, display 206, etc.) maybe used to implement any of the other computing devices and componentsdescribed herein. For example, the various components herein may beimplemented using computing devices having components such as aprocessor executing computer-executable instructions stored on acomputer-readable medium, as illustrated in FIG. 2. Some or all of theentities described herein may be software based, and may co-exist in acommon physical platform (e.g., a requesting entity can be a separatesoftware process and program from a dependent entity, both of which maybe executed as software on a common computing device). Additionally, thecomputing device 200 may include a metadata manager 201 a, which canperform the various metadata collection and generation processesdescribed herein as a replacement for, or augment to, any otherprocessor 201 that the computing device 200 may include. That is, themetadata manager 201 a may include a separate processor and/or set ofcomputer-executable instructions stored on a computer-readable mediumthat, when executed by a processor, cause the processor (or thecomputing device 200 as a whole) to perform the various metadatacollection and generation processes described herein. The metadatamanager 201 a may also include secure memory (not shown), which canstore the various criteria for collecting and generating metadatadescribed herein. The secure memory can be any desired type of memory,and can have enhanced security features to help restrict access (e.g.,can only be accessed by the metadata manager 201 a, can be internal tothe metadata manager 201 a, etc.). Where the metadata manager 201 aincludes a separate set of computer-executable instructions, theseinstructions may be secured such that only authorized users may beallowed to modify, augment, or delete them.

In some embodiments, the metadata manager 201 a may be implemented as anapplication specific integrated circuit (ASIC). That is, the metadatamanager 201 a may be a chip designed specifically for performing thevarious metadata collection and generation processes described herein.Further, the ASIC may be implemented within or in communication withvarious computing devices provided herein.

One or more aspects of the disclosure may be embodied in acomputer-usable data and/or computer-executable instructions, such as inone or more program modules, executed by one or more computers or otherdevices. Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types when executed by a processor ina computer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects of the disclosure, and such datastructures are contemplated within the scope of computer executableinstructions and computer-usable data described herein.

FIG. 3 is a diagram showing an example system architecture 300 on whichvarious features described herein may be performed. The system 300 ofFIG. 3 depicts a local office 103, a first premises 102 a, a secondpremises 102 b, one or more content databases, such as contentdistribution networks (CDN) 310 and 320, a network 330, and a secondscreen experience computing device (e.g., server) 340. As shown in FIG.1, the local office 103 may connect to the first premises 102 a andsecond premises 102 b via links 101. The first premises 102 a mayinclude an interface 120 (e.g., a gateway), a first screen device 301(e.g., a television, a monitor, a projector, a smartphone, etc.), andone or more second screen devices 302 (e.g., a smartphone, tablet,laptop, etc.). As shown, in FIG. 3, multiple users A and B may belocated at the first premises 102 a and each user may operate a secondscreen device 302 while consuming content via the first screen device301. Meanwhile, the second premises 102 b may include an interface 120,a first screen device 301, and a second screen device 302 used by a userC. Content, such as video content, may be transmitted (e.g., streamed)from the local office 103 to the interfaces 120 of the first and secondpremises 102 a-b, and to the first screen devices 301. Thus, users A andB may consume content (e.g., view the content) at the premises 102 a anduser C may consume content at the premises 102 b. Notably, whileconsuming content, each user may operate a respective second screendevice 302 to access data related to the content consumed on the firstdevice 301 at their premises 102. For example, user A may operate asecond screen device 302, such as a smartphone, to access data, such asthe name of an article of clothing worn by an actor shown in the contentstreamed through the first screen device 301. The data may be any data,such as metadata, that provides information or additional content tosupplement the primary content (e.g., linear television program,Internet or other network-stored content, on-demand movies, etc.)consumed on the first screen device 301. For example, data may include alink to an information source, such as a webpage, indicating where anarticle shown in the primary content can be purchased and how much itcan be purchased for, a video clip with bonus features, text and/orimages with information about the content itself or about individuals oritems shown in the primary content, advertisements, coupons, questionspertaining to the primary content, etc. This data may be generated byviewers, and may grow over time as more users view the content. Forexample, data may include user commentary about scenes or events in thecontent. Also, for example, the data may include commentary from auser's friend(s) regarding different scenes in a movie, and thecommentary may be tagged to points in time in the movie, so that theymay be displayed at the appropriate time. Fans may annotate a particularscene with a link to a webpage where an item in the scene may bepurchased. The various data may be collected from ordinary everydayconsumers of the content, as well as from formal content sources. Thecollection and use of this data to generate metadata will be describedfurther below.

Referring to FIG. 3, users may consume content at a premises 102 a(e.g., a home, business, etc.). Consuming content may include, forexample, watching and/or listening to a television program or anInternet (or another local or global network) video on a first screendevice 301. The first screen device 301 may receive the content from theinterface 120, which is connected to the local office 103 and configuredto retrieve the content. FIG. 3 also illustrates some examples of secondscreen devices 302, namely a smartphone and a laptop computer. Each (orsome) second screen device 302 may be configured to capture audio inconnection with the content on the first screen device and to collect,display, and communicate data in response to user inputs. The audio maybe the audio associated with the content, e.g., the soundtrack, actors'voices, or other audio signals (e.g., tones) inserted into or carried bythe primary content for purposes of audio identification. In cases whereother audio signals, such as tones or beeps, are embedded into theprimary content, those audio signals may or may not be discernible by auser, but may be detected by a second screen device 302. Further, forexample, the second screen device 302 may be a smartphone having anapplication that allows the smartphone to capture audio through themicrophone of the smartphone and respond to user inputs through a keypador touchscreen of the smartphone to obtain data related to contentconsumed on a first screen device 301. Although FIG. 3 shows someexample second screen devices 302, many other devices may be used assecond screen devices 302. Indeed, another television, similar inconfiguration to a first screen device 301, may be used as the secondscreen device 302. Moreover, it should be understood that the secondscreen device 302 might not have a screen, and could be any device, suchas a television remote controller, that has input functionality.

Further, each of the second screen devices 302 may be configured tobi-directionally communicate via a wired and/or wireless connection withthe second screen experience computing device 340 via the network 330.Specifically, the second screen devices 302 may be configured to accessthe network 330 (e.g., the Internet or any other local or wide areanetwork, either public or private) to obtain data and totransmit/receive the data via the network 330 to/from the second screenexperience computing device 340. For example, a second screen device 302may transmit data through a wired connection, including the links 101through which the content is supplied to a first screen device 301, tothe local office 103 which then routes the transmission to the network330 so that it may eventually reach the second screen experiencecomputing device 340. That is, the second screen device 302 may connectto the interface 120 and communicate with the second screen experiencecomputing device 340 over-the-top of the links 101 used to transmit thecontent downstream. Alternatively, the second screen devices 302 mayconnect directly to the network 330 to communicate with the secondscreen experience computing device 340. For example, a second screendevice 302 may wirelessly communicate using, for example, a WiFiconnection and/or cellular backhaul, to connect to the network 330(e.g., the Internet) and ultimately to the second screen experiencecomputing device 340. Accordingly, although not shown, the network 330may include cell towers and/or wireless routers for communicating withthe second screen devices 302.

Although FIG. 3 depicts the second screen experience computing device340 as being separate from the local office 103, in some embodiments,the second screen experience computing device 340 may be located at thelocal office 103. In such embodiments, the second screen devices 302 maystill access the second screen experience computing device 340 throughthe network 330. Further, even though the second screen experiencecomputing device 340 is shown as a single element, in some embodiments,it may include a number of computing devices 200.

Still referring to FIG. 3, the local office may include a router 305, asecond screen experience management platform 306 for executing any ofthe steps described herein, and a database 307 for storing userinformation (e.g., user profiles), audio files, metadata, and/orcomputer-executable instructions for executing audio recognitionprocesses or any of the steps described herein. The router 305 of thelocal office 103 may forward requests for content from users and/or userdevices (e.g., display device 112) at premises 102 to one or more CDNs310 and 320 that may supply the requested content. Each of the CDNs 310and 320 may include one or more routers 311 and 321, whose purpose is toreceive requests from users (e.g., via their local offices) and routethem to servers within its network that may store the requested contentand be able to supply it in response to the request. A CDN 310 for agiven piece of content might have a hierarchy of one primary source, anda plurality of lower-level servers that can store (e.g., cache) thecontent and respond to requests. The lower-level servers that ultimatelyservice the request may be referred to as edge servers, such as one ormore edge servers 312 and 322. The various servers may include one ormore content databases 313 and 323, which store content that therespective CDN 310 and 320 manages. In some embodiments, the CDNs 310and 320 may provide the same or similar content. In other embodiments,the content of the CDNs 310 and 320 may offer different content from oneanother. Also, the CDNs 310 and 320 may be maintained/operated by thesame or different content providers. Although only two CDNs 310 and 320are shown, many CDNs may be included in the system architecture 300 ofFIG. 3.

FIGS. 4A-4H are diagrams illustrating example screens of an application(or program) configured to allow users to create and/or view metadatarelating to a program or content they are consuming. The screens inFIGS. 4A-4H may be displayed on a second screen device 302. A user mayoperate his/her second screen device 302 to start an application, whichmay render one or more of the screens shown in FIGS. 4A-4H.

In some cases, the user may have to log-in to proceed to use one or moreof the features of the application. As shown in FIG. 4A, logging-in mayrequire entering a username and/or password. In this manner, theapplication may identify a user of the second screen device 302 runningthe application. Once the user is logged-in, actions, such as enteringdata for tag submissions and/or editing pre-existing metadata tags, maybe automatically associated with the user.

FIG. 4B illustrates a screen of the application in which a user may edithis/her profile. The profile may be used to customize filters which mayfilter metadata displayed to the user. Various items may be set in theuser profile, such as the user's age, interests, favorite music, etc.Based on this information supplied by the user, only certain metadatamay be shown thereby improving the user's second screen experiencewithout overwhelming the user with an excessive amount of metadata thatmay exist. For example, if a user likes sports, metadata related tosports may be shown on the user's second screen device 302 while othermetadata may be filtered out. By supplying profile information, usersmay also be able to see metadata from other users with similarinterests. For example, a user who indicates that she likes countrymusic may choose to specifically receive metadata created by other userswho have indicated in their profile that they like country music.

Further, FIG. 4C illustrates a screen that allows a user tocontrol/customize the filters themselves. As shown, the application mayallow a user to filter the metadata received to include only metadatafrom certain individuals (e.g., friends). For example, a user may bemore interested in what metadata their friends have submitted than inthe metadata created by others. Therefore, the user may control a filterto specify which people (e.g., friends, family, celebrities, etc.)he/she would like to see metadata from.

Another example filter shown in FIG. 4C may allow a user to specify thetypes of metadata that he/she will receive. For example, if a userdesires only to view metadata that is in the form of a video (e.g.,bonus video), the user may specify this in a filter of the application.Yet another filter may allow the user to filter the metadata by itsgenre. As explained herein, when metadata is entered, a genre may bespecified for the metadata. This specified genre may be a basis on whichother users filter the metadata they wish to view. An example of a genreof metadata may be “trivia.” Specifically, some metadata may becharacterized as trivia questions pertaining to aspects of the contentassociated with the metadata. For example, if an actor enters a scene,metadata of the “trivia” genre may include a question such as “Do youknow where the actor is from?” and/or a statement indicating that theactor is from “Barcelona, Spain.”

FIGS. 4A-4C show example screens for configuring the application for aspecific user. In light of these example screens, it should beunderstood that various other screens may be used to configure theapplication. Moreover, users may choose to configure the application atany time and/or in various orders. For example, a user may access ascreen like the screen in FIG. 4C to configure a filter before accessinga screen like the screen in FIG. 4B to configure a profile.

In any event, the user may eventually view a screen like the one shownin FIG. 4D. As shown in the screen of FIG. 4D, a user may choose toselect a process for identifying content that the user is currentlyconsuming. Identifying the content allows the system to provide the userwith the correct stream of metadata, and also helps to match any datauploaded by the user to the correct program and time within the programso that metadata tags may be generated. In some embodiments, theidentification can be done with a simple exchange of informationbetween, for example, the second screen device 302 and a primary device,such as a STB or interface 120. For example, the STB may simply reportto the device 302 the channel number or service identifier currentlybeing displayed on the main screen 301, and the current playback time.However, in some embodiments, the interface 120 might be a legacy devicethat lacks the ability to directly communicate with the second screendevice 302 in this manner. In such situations, the second screen device302 may detect an audio profile or capture an audio sample of thecontent being consumed, and the audio profile or audio sample may beused to identify the content being consumed and the current playbacktime within the content. For such an embodiment, in response to aselection of the “Identify Content” button in FIG. 4D, a recordingscreen similar to the screen in FIG. 4E may be displayed on the secondscreen device 302. Specifically, the screen in FIG. 4E may be displayedwhile the application detects an audio profile or records an audio clip.When the second screen device 302 is in proximity to a first screendevice 301, the detected audio profile may represent the audioassociated with the content streaming via the first screen device 301.Once the audio profile is detected, the application may determine theidentity of the content streaming on the first screen device 301 basedon the audio profile. This determination may include transmitting theaudio profile to another device, e.g., the second screen experiencecomputing device 340, and receiving a message indicating the identity ofthe content. The second screen experience computing device 340 mayperform audio recognition techniques to identify the content. Suchtechniques may include comparing the audio profile with recorded audiosamples or other audio profiles from all of the various contentofferings made available to the user, to identify an audio match. Theserecorded audio samples and/or other audio profiles used for thecomparison may be stored in databases within the second screenexperience computing device 340 or elsewhere (e.g., in the local office103 or in other computing devices 200 connected to the network 330). Insome examples, the search for matching recorded audio samples or otheraudio profiles may be narrowed with the assistance of electronic programguides. For example, the second screen experience computing device 340may consult one or more electronic program guides to determine whichrecorded audio samples or audio profiles to use for the comparison withaudio profile received from the second screen device 302.

The identification described above is initiated by the user selectingthe “Identify Content” button, but the identification need not requiresuch a user initiation. For example, in some embodiments, theidentification process can automatically occur whenever the userlaunches the metadata application on the second screen device 302, orwhenever the user enters data that he/she would like to upload for thecontent that he/she is currently watching, or when the user wishes totag a point in a program for which he/she will eventually upload acomment or data (e.g., if the user needs time to collect Internet linksfor the comment, or to draft the comment, the user can simply tag thepoint in time and then subsequently draft the data to be associated withthe tagged point in time in the content).

When the application ultimately determines the identity of the content,a screen similar to that shown in FIG. 4F may be displayed so that theidentity of the content may be shared with the user of the second screendevice 302. Moreover, the screen in FIG. 4F may indicate a data tagidentifier (e.g., “Tag 1”) and a time stamp (e.g., 12 minutes and 25seconds) identifying a time within the identified content thatcorresponds to the audio profile or audio clip. From the screen in FIG.4F, a user may select to enter data related to the tag. Once the data isentered, a user may submit a metadata tag submission to another deviceand/or service that is responsible for collecting data from users (e.g.,the second screen experience computing device 340 or another computingdevice including a metadata manager 201 a) and generating a metadata tagbased on the collected data. Additionally, the screen in FIG. 4F mayalso allow a user to remove tags if the user later decides not to submitthe data.

FIG. 4G illustrates a screen that displays a metadata entry formconfigured to facilitate entry of data to be included in a metadata tagsubmission. The form may be generated and displayed automatically inresponse to identifying the content provided on the first screen device301 from the audio profile or audio clip or in response to a userselection. As shown in FIG. 4G, the form may include a number of fields401 in which data may be entered. Although FIG. 4G shows that each fieldmay be for a different type of data, in other cases one field may beconfigured to receive any type of data. In addition to the fields, forentering data, a separate genre field 402 may exist for giving the userthe option to classify the data he/she is entering by genre. Forexample, the user may select from a drop down menu a genre indicatingthat the data she has or will enter may be classified as biographicalinformation related to the associated content. Also, beside each of thefields 401, the form may include a link for triggering the generation ofa dialog box. For example, when a user selects the “Attach” link next tothe field for the web link, a dialog box 403 may be displayed.

FIG. 4H illustrates an example dialog box 403 that may appear over themetadata entry form. The dialog box 403 may allow a user to browse forthe data. For example, the dialog box 403 may allow a user to browsefiles stored on the hard drive of the second screen device 302 or on alocal network (e.g., a local media server). Alternatively, the dialogbox 403 may function like a web browser to allow a user to navigate to awebsite and select a URL of the website, a link within the website, orany other object within the website. The dialog box 403 may allow such aselection to be imported into a field 404 within the dialog box 403.Then, after the user selects “submit” in the dialog box, the data in thefield 404 of the dialog box 403 may be imported into the appropriatefield 401 of the form. Finally, the data entry may be completed when theuser selects “submit” in the metadata entry form. In response toselecting “submit” in the metadata entry form, the application maygenerate a metadata tag submission described herein. Further, althoughnot depicted in the drawings, after selecting “submit” in the metadataentry form, the application may render a screen showing the submitteddata or a screen showing other data submitted for the content providedon the first screen device 301. Additionally, it should be understoodthat multiple types of data may be entered into the fields 401 when“submit” is selected in the metadata entry form so that multiple typesof metadata tags may be generated for a similar time point of thecontent provided on the first screen device 301. For example, a user mayinsert text indicating that she bought the same shirt as a particularactress and a web link where it can be purchased.

The metadata entry forms shown in FIGS. 4G and 4H are just examples. Insome embodiments, the metadata entry forms may include specific fieldsthat require a user to enter specific information so that each metadatatag created may have a similar format. For example, a user may berequired to select a character, an item of clothing, and store fromvarious drop down menus in order to submit a metadata tag submission.While such an embodiment may hinder a user's creativity, the structuredmetadata tag submission may be beneficial when filtering the metadata.

In addition, the metadata entry form may include a field (not shown) fordesignating target individuals (e.g., friends, family members, etc.) sothat the tag submissions may be available exclusively for the targetindividuals. For example, a user may select one or more friends thathe/she would like to share the metadata with so that when the tagsubmission is sent, it will only be available to the selected one ormore friends. When one or more of the selected friends later views theassociated content, he/she can see the metadata in the tag submissionthat was made available to them. Thus, the metadata entry form may beused to filter metadata submissions on the upstream side as well.

Although the above description explains that the screens in FIGS. 4A-4Cmay belong to an application, it should be understood that the screensmay also be webpages of a website displayed by a web browser. That is,in some embodiments, a user may navigate to a designated website andsubmit metadata tag submissions through the website.

FIG. 5 is a flow diagram illustrating an example method of the presentdisclosure in which a user may generate and upload metadata relating tocontent that he/she is consuming. In particular, FIG. 5 describes anexample process of acquiring data from a user for uploading to thesecond screen experience computing device 340. As explained above, anaspect of the present disclosure is to provide a method that allowsmultiple users (e.g., the “crowd”) to assist in generating second screenmetadata so that other users viewing video content at a later time mayenjoy an enhanced experience by accessing the metadata generated by thecrowd. The process in FIG. 5 illustrates how data to be included in ametadata tag submission may be acquired from a single user. It should beunderstood that many users may perform a similar process as illustratedin FIG. 5 so that a large amount of metadata may be obtained. Also, withregards to the description related to FIG. 5, where the disclosurerefers to steps performed by a second screen device 302, it should beunderstood that these steps may be performed by a computing deviceprocessor, such as a processor in second screen device 302, executingcomputer-executable instructions stored on the second screen device 302.Alternatively, the steps may be performed by any other device that theuser may use to generate and/or view metadata for a piece of content.

As shown in FIG. 5, the process may begin with step 501 in which contentis provided to the user for consumption, such as via file-basedtransfer, unicast and/or multicast streaming, analog or digitalbroadcasting, playback of previously-stored content (e.g., contentrecorded by a DVR or downloaded at an earlier time), etc. Referring toFIG. 3, step 501 may entail video content being supplied from one ormore of the CDNs 310 or 320 to the local office and downstream to one ormore of the premises 102. At the premises 102, the video content may bereceived through the interface 120 and streamed through the first screendevice 301. In short, step 501 may include, for example, the known stepsfor delivering for consumption an item of content to a display device.Additionally, delivering content in step 501 may include deliveringaudio associated with the video content and/or audible features designedto identify timing within the content. The audio content may beoutputted via the first screen device 301 itself or another deviceconnected to the first screen device (e.g., a speaker system).

In step 502, a user may decide that he/she would like to create somesupplemental information or content for a particular scene or point inthe content. For example, the user may wish to alert others that aparticular piece of art in the background was created by a friend. Toinitiate the supplemental content generation, the user may first enteran input to tag the point in time in the primary content. This may bedone, for example, by pressing the “Identify Content” button discussedabove. The input may be received via a second screen device 302.Specifically, the user input received at step 502 may be an instructionto generate a tag to identify a point in the content (e.g., a point in atelevision program) with which the user wishes to associate information.Inputs may be made by users in various ways, such as pressing a buttonon a keypad of the second screen device 302, pressing a virtual buttonon a touch-screen of the second screen device 302, submitting a voicecommand to the second screen device 302, making a predetermined gestureor body movement detected by a camera, etc.

The time at which a user input is received at step 502 depends on theuser. When the user consumes (e.g., views) an event on the first screendevice 301 and decides to create metadata related to the event, the usermay enter user input. For example, if the user is watching a televisionshow and an actor appears on the first screen device 301 wearing asweater, the user may decide to create metadata that specifies where thesweater may be purchased, and therefore, may enter user input at step502 to trigger metadata creation. Herein, an event may refer to anyoccurrence, such as a playback point in time, a scene, a chapter, acharacter's appearance, etc., within the content streamed on a firstscreen device 301. At any given time point there may be multiple events.Further, different users may perceive different events, and thus,different users may choose to create metadata for different eventsoccurring at the same time within the content streaming on the firstscreen device 301. By allowing users to dictate the time at which theycan enter data, a large amount of metadata may be acquired andorganized.

In response to the user input at step 502, an audio sampling (or audioclip) may be captured at step 503 to help identify the program and aportion of the program to be associated with the user's data. Forexample, the second screen device 302 may detect and process a 15-secondsegment of the audio portion of the content and generate a data profileor fingerprint of the detected audio. Specifically, an application onthe second screen device 302 may use one or more audio fingerprintingtechniques to generate the data profile or fingerprint. The profile ofthe audio portion may identify detected characteristics of the sound,such as frequencies sampled, volume level, times at which certainfrequencies or volume levels were detected, etc. The purpose of theaudio profile is to provide data from which an identity of the contentbeing consumed by the user (e.g., streaming on the first screen device301), as well as a point in time within the content, may be obtained.Notably, it might not be necessary for the first screen device 301 orthe interface 120 to send any information other than the sound to thesecond screen device 302 in order for the second screen device 302 toidentify the content being streamed, thereby allowing operation withlegacy devices.

In step 503, the second screen device 302 which receives the user inputmay detect audio through its microphone in response to detecting theuser input. Where the content being consumed or rendered on the firstscreen device 301 includes audio, this audio may be detected by a secondscreen device 302 that is in relatively close proximity to the firstscreen device 301 (e.g., within a range that the audio from the contentmay be detected). For example, while watching a television program on afirst screen device 301, a user may operate a second screen device 302to detect audio associated with the content displayed on the firstscreen device 301, such as speech of actors within the televisionprogram, background music of the television program, etc. Of course, thesecond screen device 302 may also detect audio from other sources notplaying the audio associated with the content on the first screen device301, such as other people talking in a nearby room. However, audio notpertaining to the content (e.g., noise) may be filtered out orheuristics may be used to analyze the audio clip so that suchundesirable noise may be neglected.

To detect the audio at step 503, the second screen device 302 may detectaudio for a predetermined period of time (e.g., for five seconds). Thatis, once a user input is detected in step 502, the second screen device302 may activate an audio profiling module or fingerprinting module,which may begin to detect audio and store information identifying theprofile of the detected audio for the predetermined time period. Thepredetermined time period may be different for different types ofcontent and in different embodiments. For example, the predeterminedtime period may be adjusted if the profile for the audio portion is tooshort to accurately identify the content streamed on the first screen301, the device 302 may extend the predetermined time period. Further,in some embodiments, the time period for processing may vary dependingon how long it takes the detecting device (e.g., the second screendevice 302) to recognize that a significant audio sample has beendetected and profiled. For example, in some cases, a second screendevice 302 may determine that it should process a ten second audioportion, while in other cases the second screen device 302 may determinethat it should process a fifteen second audio portion (e.g., where thefirst 5 seconds of recording were silent). Still, in some embodiments,the duration of the audio portion may be relatively constant, and if theaudio portion is not sufficient to identify the content, processing theaudio may be repeated a number of times or until the content isidentified. The second screen device 302 may send the audio profile toanother device (e.g., the second screen experience computing device 340)to determine if it is sufficient to identify the content. Then, based ona response from the other device, the second screen device 302 maydetermine whether or not to process another audio portion. If the otherdevice is unable to identify the content or the time point within thecontent that the audio profile pertains to, the second screen device 302may send an audio profile identifying characteristics of the audiosignals detected immediately after the insufficient audio profile. Inthis manner, the user may still associate metadata with a desired pointin time of the content. To accomplish this, the second screen device 302may process a number of audio portions or a certain amount of audiosignals of the content. In some embodiments, the second screen device302 may use a buffer to implement a first-in-first-out (FIFO) queuestoring actual audio clips or audio signals of the profiled portions, sothat audio clips or audio signals are temporarily stored. The audioclips or audio signals may be dumped after a certain period of time ordepending on the size of the buffer. This buffering, however, isoptional, as the audio profiling or fingerprinting application maysimply generate the audio profile data dynamically as audio is detected,without requiring recording the audio.

While FIG. 5 illustrates that the user input at step 502 is performedprior to detecting the audio at step 503, it should be understood thatthis is an example embodiment. In some embodiments, the audio may bedetected prior to receiving a user input. For example, a second screendevice 302 may begin processing audio when it is first powered on orwhen an application on the second screen device 302 is initiallyexecuted. Also, the second screen device 302 may intermittently (e.g.,periodically) process audio or may continuously process audio as long asit is on or as long as an application on the second screen device isrunning. By intermittently or continuously processing the audio, theapplication may ensure synchronization of the metadata with the contentprovided on the first screen device 301. Further, where the audio isdetected prior to receiving an input, when the user does make the input,the input may trigger a process of identifying a time point andcapturing metadata for that time point. For example, the process maygenerate an audio profile representing a time point of the content andproceed to step 504.

Furthermore, in some embodiments, where audio of content provided on afirst screen device 301 is detected prior to receiving a user input onthe second screen device 302, the user input may also trigger thecontent on the first screen device 301 to pause. For example, thecontent provided on the first screen device 301 may be video-on-demandcontent, which may be paused. To allow users to enter metadata withoutmissing the content streaming on the first screen device 301, theapplication on the second screen device 302 may automatically pause thecontent in response to receiving a user input to begin entry of usermetadata. Specifically, the second screen device 302 may pause thecontent provided on the first screen device 301 by transmitting a signal(e.g., direct infrared signal to a set-top box, or indirect signal bysending a packet to an Internet site that, in turn, transmits a pausecommand to a digital video recorder or set-top box using EBIF—EnhancedTV Binary Interchange Format—messaging) to the first screen device 301or another device (e.g., a set top box) associated with the first screendevice 301.

In step 504, a user may operate the second screen device 302 to enterthe data or information that the user wishes to upload. Entering datamay include various processes. For example, it may include typinginformation, such as the user's commentary, or the name of a store whichsells a particular product displayed on the first screen device 301 orthe webpage of said store. Further, multiple types of data may beobtained at step 504. For example, a user may acquire both a link for awebpage and text in one implementation of step 504.

In some embodiments, entering data may include navigating to a webpageand copying a link for the webpage and then pasting the link into a dataentry area on the device 302. A user may be able to navigate to thewebpage from within the same application that is used to record theaudio file and acquire the data so that the user can simply select abutton to import the universal resource locator (URL) of the webpage asthe data. Alternatively, the application on the second screen device 302may launch a separate application, such as a web browser, to navigate anetwork, such as the Internet, to locate data and then press a button tocapture the URL from the web browser. Therefore, instead of having toperform known copy and paste functions to obtain a URL, the applicationused to acquire data may include a function that captures a URLautomatically.

The user may also use his/her second screen device 302 to create datathat is entered. For example, the user may use a global position system(GPS) receiver of the second screen device 302 to identify his/herlocation and submit the location information as data. The locationinformation may indicate where a product in the content may be found orthe geographical location of a scene within the content. Similarly, theuser may also use a camera or microphone of the second screen device 302to capture an image or sound bite that may be used as data to be enteredand included in the metadata tag submission.

The entered data may be correlated with the audio profile in step 505.This correlation may be performed automatically by the second screendevice 302. For example, after the audio profile is generated in step503, an application on the second screen device 302 may prompt the userto enter the data, and thus, when the user enters the data, the entereddata may be automatically correlated with the audio profile mostrecently generated. In other examples, a user may input data and thenchoose to correlate the data with an audio profile previously detected.Thus, a user may generate an audio profile while watching a televisionprogram and may later enter data and correlate the data with the audioprofile after the television program is over or during a commercialbreak.

The application may alert the user that there are audio profiles forwhich metadata tag submissions have not been created. For example, theapplication may highlight a portion of the display of the second screendevice 302 to indicate that the user has incomplete metadata tagsubmissions outstanding. In some cases, the application may evenindicate how many incomplete metadata tag submissions the user haspending.

After the data and audio profile are associated with one another, thesecond screen device 302 may generate, or provide information to anotherdevice to generate, a tag submission at step 506. The tag submission maybe a packet, such as an Internet Protocol version four (IPv4) orInternet Protocol version six (IPv6) packet. The packet may include aspecific header that addresses the packet to a designated server, suchas the second screen experience computing device 340. The packet mayfurther include combined data including the audio profile and the data.In some examples, this combined data may be compressed so that the datain the packet does not include all of the data for the audio profileand/or metadata. In particular, where an audio clip rather than an audioprofile is to be included in the packet, the audio clip may becompressed to form data that adequately represents the audio clip.Further, the packet may or may not be stored. The packet may begenerated without being stored so as to preserve memory on the secondscreen device 302.

Once the tag submission is generated, the tag submission is transmittedat step 507 to the destination indicated by the header of the tagsubmission, for example, the second screen experience computing device340. In some embodiments, the tag submission may be transmitted in theupstream bandwidth on the same links 101 used to receive the contentstreamed on the first screen device 301. That is, the tag submission maybe transmitted via the links 101 through which content is sentdownstream from the local office 103 to the premises 102. In this case,the second screen device 302 may first wirelessly transmit the tagsubmission to the interface 120 which in turn may direct the tagsubmission upstream. Alternatively, the second screen device 302 maytransmit the tag submission over a wireless network, such as a cellularbackhaul, or over another wired connection used by the second screendevice 302 for connecting to the network 330 (e.g., the Internet oranother local or wide area network).

At step 508 the process of FIG. 5 may determine whether another userinput to tag the content for metadata upload is received. If anotheruser input is received, the process returns to step 503 to detectanother audio profile. Accordingly, steps of FIG. 5 may be repeated togenerate multiple tag submissions including data for various segments ofcontent streamed on the first screen device 301. Notably, the contentbeing streamed may change, and thus, the audio profile in a subsequentimplementation of step 503 may be for different content than theprevious implementation of step 503. Meanwhile, if a new user input isnot detected, step 509 may be performed to determine if a wait periodhas elapsed since the last user input or whether an application forentering the data is still running. If the wait period has elapsed orthe application is terminated, the process of FIG. 5 may end. Otherwise,step 508 may be continuously performed until a new user input isdetected. In the meantime, other steps may also be performed. Forexample, the application on the second screen device 302 may change itsdisplay from displaying a form for entering data to a screen for viewingmetadata created by others.

In some embodiments, the second screen device 302 may identify thecontent itself based on the audio profile, and may determine a timepoint for the audio profile. Or the time point may be obtained from theprimary device (e.g., STB displaying content on the display 301). Insuch cases, the tag submission might not include the audio profile (andthe device 302 might not bother detecting one), and instead may includeidentification information identifying the content, the time point ofthe audio profile, and secondary information (e.g., the data entered bythe user). Still, in other embodiments, the second screen device 302 maysend the audio profile to a server (e.g., the second screen experiencecomputing device 340), and may receive identification information fromthe server in response (e.g., information identifying the content and/orinformation identifying a time point of the audio profile, which thecomputing device 340 may determine by comparing the audio profile withpre-recorded samples or pre-configured profiles of known programs). Inthese embodiments, the tag submission also might not include the audioprofile, and instead may include the received identification informationand corresponding secondary information (e.g., data).

Next, an example method of processing metadata tag submissions isdescribed with reference to FIG. 6. As explained above, a second screenexperience computing device 340 may be configured to store metadata andtransfer the metadata to second screen devices 302 so that users mayenjoy an enhanced experience. FIG. 6 illustrates how metadata tags maybe received by and stored within such a second screen experiencecomputing device 340. FIG. 6 begins with step 601 in which a tagsubmission is received. The tag submission received in step 601 maycorrespond to tag submissions transmitted in step 507 of FIG. 5, or maybe received or accessed from another source.

In some examples, step 601 may include decoding or parsing the tagsubmission. As discussed above, the tag submission may include an audioprofile and associated data. Thus, when receiving the tag submission,the second screen experience computing device 340 may decode the tagsubmission (e.g., an IP packet) and separate the data related to theaudio profile from the data to be used for the metadata. Similarly,other information included in the tag submission (e.g., an IP packetheader) may also be distinguished from the audio profile data and data.In some embodiments, the tag submission may be analyzed to determinewhether a time and content identifier is included and/or whether anaudio profile is included.

Further, it is contemplated that a computing device, such as the secondscreen experience computing device 340, may receive multiple tagsubmissions. In some examples, the second screen experience computingdevice 340 may be configured to receive more than one tag submission atthe same time. To facilitate the potentially large volume of tagsubmissions that may be received, the second screen experience computingdevice 340 may include multiple computing devices (e.g., servers).

In step 602, the second screen experience computing device 340 mayidentify the content associated with the tag submission. Specifically,the audio profile data from the tag submission may be analyzed toidentify the content that was consumed through the first screen device301 by the user who triggered generation of the tag submission.Analyzing the audio profile data may include performing any audiorecognition process. To identify the content represented by the audioprofile, the second screen experience computing device 340 may includeor have access to a database including actual audio files or audioprofiles for various content. In some examples, such a database mayinclude audio files and/or audio profiles for all content that maypossibly have been consumed by users. In performing an audio recognitionprocess, the second screen experience computing device 340 may accessone or more electronic program guides to facilitate (e.g., by narrowinga list of possible audio files based on availability of content) asearch for a matching audio file within the database. In some cases,such a database might not include audio files and/or audio profiles forall possible content, and therefore, the content might not beidentified. Notably, in some embodiments, audio recognition processesmay be performed by the second screen device 302, and thus, the processof step 602 at the second screen experience computing device 302 may beoptional.

Step 603 may determine whether the content has been identified in step602. If the content is identified (Yes at step 603), the method mayproceed to step 604. If the content is not identified (No at step 603),a message may be transmitted to the second screen device 302 that sentthe tag submission, which included the content that was not identified,notifying the sender that the content could not be identified and themethod of FIG. 6 may thereafter end. In response to receiving themessage, the user may choose to resubmit the tag submission or respondto the message with additional information that may aid in identifyingthe content (e.g., the name of a television program).

In step 604, one or more filters may optionally be applied to the data.Specifically, the data may be filtered to determine whether the user'suploaded data should be added to the body of metadata that is stored forthe particular piece of content. For example, a filter may be applied todetermine whether data is appropriate (e.g., whether data includesundesirable language or images). In some examples, whether data isappropriate may vary depending on a rating (e.g., G, PG, PG-13, R, etc.)of the identified content. Therefore, filters for G-rated content mayflag any data that includes any “bad” language or images, which would gounflagged through filters for R-rated content. Another example of afilter may include a filter to determine whether the data is relevant.That is, filters may be generated to analyze the data to determinewhether it pertains to the content that it purports to be related to.Such a filter may search the data for words that match a list ofexpected words or similar words to other metadata for the same contentthat has already been supplied. Additionally, or alternatively, such afilter may analyze the data to determine whether it is known spam. Yetanother filter may determine whether the data is a duplicate. It iscontemplated that many users may submit data, and therefore, it isrecognized that users may submit the same data. To avoid or reduce theoccurrence of duplicate metadata, the second screen experience computingdevice 340 may implement a filter that analyzes previously and/orsimultaneously submitted data to determine if the data is a duplicate.Duplicate data may also be the result of multiple occurrences of thesame event within the same content. For example, an actress may wear thesame shoes in different scenes, and thus, users may submit the same datafor different scenes. The filter used to prevent or reduce duplicatesmay distinguish whether the duplicate is for the same segment ofcontent. For example, the filter may allow duplicate data when the datapertains to different segments, but may flag duplicate data that is forthe same segment of the same content. In some embodiments, where filtersflag certain data, one or more authorized personnel may be tasked withreviewing the flagged data and deciding whether to remove or keep theflags. In other words, there may be a task force or team of people forreviewing flagged data to determine which data is ultimately stored asmetadata.

The above described filters are just a few examples of the many filtersthat may be applied to the data to control which metadata may be madeaccessible to second screen devices 302. Further, the present disclosureexplains that filters may flag certain data. Flagging data may includesetting a value (e.g., a flag) of a memory cell associated with the datato ‘1.’ Alternatively, instead of flagging data, the filters may deletethe data from a storage area (e.g., a cache) where the data that isreceived in step 601 may be temporarily stored.

Next, at step 605, the second screen experience computing device 340 maydetermine whether a metadata tag submission should be accepted intostorage for storing metadata based on the results of the filtering.Where flags are used to mark data, determining whether a metadata tagsubmission should be accepted may comprise checking whether flags existfor the data. For example, if the data is flagged, then thecorresponding metadata tag might not be stored. Alternatively, wherefilters delete certain data that does not pass, then determining whetherthe metadata tag submission should be accepted may be performed bydetermining if the associated data still exists in a temporary storagearea. If the metadata tag submission is not accepted (No at step 605),the method may end. In this case, the metadata tag corresponding to themetadata tag submission received at step 601 might not be stored on thesecond screen experience computing device 340. In some embodiments, thesender of the tag submission may be notified that the tag submission wasdenied in a similar manner that a notification is sent when contentcannot be identified.

On the other hand, if the tag submission is accepted, e.g., the datapasses the filtering (Yes at step 605), the method may proceed to step606. In step 606, the corresponding metadata tag may be stored so thatthe content identified in step 602 may be tagged with the data receivedin the tag submission. Storing the metadata tag may include storing thedata in association with a time point of the identified content. Thetime point may correspond to the time within the content that matchesthe audio profile that was included within the submission. For example,if the audio profile represents a five second audio clip occurring tenminutes into a television program, the metadata associated with thataudio profile may be associated with a time point referencing theten-minute mark of the television program. In some examples, whendetermining the time point of the content to tag, the method mayconsider that there may be a delay between the time when the userconsumes an event and when the user triggers the generation of the audioprofile. In recognition of such a delay, the method may tag the contentat a point in time preceding the time represented by the audio profileby a predetermined interval (e.g., ten seconds). Alternatively, insteadof associating the tag with a specific time point, the tag may beassociated with a segment of the content. For example, a half-hourtelevision show may be divided into six five-minute time segments, andif the audio profile is determined to represent a time point of eightminutes into the television show then the tag may be associated with thesecond segment (e.g., the segment between five minutes and ten minutes).

In some embodiments, the second screen experience computing device 340may transmit a confirmation message to the user who sent the metadatatag submission to ask the user to confirm the time point of the metadatatag submission. In response to the confirmation message, a user maymodify the time point thereby fine-tuning the association of themetadata to the time point in the content. The user may also modify theinformation in the metadata tag. In some examples, the confirmationmessage may include an image showing a screen shot (or frame) from thecontent at the time point that the metadata tag will be associated with.Thus, a user may view the image and determine whether the time point ofthe metadata tag is correct. Alternatively, or additionally, theconfirmation message may include an audio clip corresponding to theaudio profile that was received so that the user may listen to the audioclip to determine whether the time point of the metadata tag is correct.

In some embodiments, storing a metadata tag at step 606 may includeediting a previous metadata tag. For example, if the time point of themetadata tag to be generated coincides with a pre-existing metadata tag,then the method may modify the pre-existing tag to include the newlysubmitted metadata. That is, editing or modifying a tag may includecombining metadata from a pre-existing tag and the metadata mostrecently acquired and associating the combined metadata with the sametime point in the content.

Additionally, the method may also include step 607 for ranking themetadata tags. In some embodiments, one or more of the metadata tags forthe identified content may be ranked so that higher ranked metadata tagsare displayed on a second screen device 302 before lower ranked metadatatags. In some examples, metadata tags are only ranked if more than onemetadata tag exists for the same time point or segment in a particularpiece of content. In this case, the ranking of the metadata tags may berelative among the metadata tags that pertain to the same time point orsegment.

Ranking may be based on the user who is responsible for generating themetadata tag. In particular, the ranking may be based on a user'sprevious metadata tag submissions. For example, a metadata tag may beranked higher because the user who generated the metadata tag submissionpreviously sent a plurality of metadata tag submissions that wereapproved. Alternatively, or additionally, the ranking may be based on auser's profile. Certain users may be more credible than others, andtherefore, metadata tag submissions from such users may be rankedhigher. Also, user profiles may indicate that certain users have beengiven priority. For example, users associated/affiliated with a contentprovider or content creator may be given priority such that metadatatags created by such users are given higher rankings. User profiles mayassist in identifying which users are affiliated with a content provideror content creator so that such prioritization scheme can beimplemented. Metadata tags from advertisers and marketing companies mayalso be given higher priority. Similarly, metadata tags created byselect individuals (e.g., friends, family, celebrities, etc.), who maybe identified as described above, may be given higher priority as well.

In some embodiments, ranking of metadata tags may be done by other usersviewing the metadata tags on second screen devices 302. Specifically,users may assist in ranking the metadata tags by rating the metadatatags. If a user “likes” a metadata tag, the metadata tag's ranking mayimprove, whereas if a user “dislikes” a metadata tag, the metadata tag'sranking may be lowered. In this manner, the crowd may be used to rankthe metadata tags so that more relevant, desirable, and/or helpfulmetadata tags are ranked higher than others.

The above methods describe example methods for leveraging individualusers, or a crowd of users, to acquire supplemental information andgenerating metadata tags based on the supplemental information. Thefollowing provides examples as to how these metadata tags may beprovided to and displayed on second screen devices 302. An examplemethod is described with reference to FIG. 7.

FIG. 7 begins with step 701 in which content is delivered forpresentation (or otherwise provided) to a first screen device 301. Whencontent is delivered or accessed for the first time, there might not beany metadata tags. However, when the content is delivered again (e.g.,when the content is video on demand (VOD) content, a rerun, or otherwisetime shifted), metadata tags generated by previous users of the contentmay be available for users to consume (e.g., view) with second screendevices 302. So, for example, a user may load or access supplementalinformation on a second screen device 302, where the supplementalinformation comprises the various user comments and metadata that havebeen entered and uploaded for the content that the user is about towatch. FIG. 8, discussed further below, illustrates an example of this.Thus, when content is delivered or accessed at step 701 after havingbeen previously provided at least once, metadata tags associated withthe content may be obtained. The more times content is streamed, themore likely it may be that people have provided metadata for thecontent. Further, in some cases, content creators, content providers, orother authorized persons or organizations may generate some metadata forcontent before it is made publically available, so that users consumingthe content for the first time may also have supplemental information toaccess with their second screen devices 302. That is, metadata may beavailable before the content is delivered for the first time, and usersconsuming the content may add to or modify what is already available.

While content is delivered in step 701, a user may power up a secondscreen device 302 and initiate an application on the second screendevice 302 that is configured to allow the second screen device 302 toreceive metadata from the second screen experience computing device 340(or another device). Specifically, a processor of the second screendevice 302 may run the application. The application may generate a userinterface on the second screen device 302 to allow a user to input acommand at step 702 when the user of the second screen device 302 wishesto view supplemental information. In response to receiving the userinput in step 702, the second screen device 302 may detect and generatean audio profile at step 703 of the content being delivered to the firstscreen device 301 so that metadata related to the content may beretrieved without the user having to provide other information toidentify the content. In addition to supplying a command at step 702, auser may also specify the types, genre, authors, etc. of metadata tagsthat he/she desires to view. Steps 702 and 703 may, in one aspect, beperformed in a similar manner as steps 502 and 503 of FIG. 5,respectively to generate an audio profile representing the content beingconsumed.

Next, at step 704, a request for metadata may be transmitted. Forexample, the second screen device 302 may generate and transmit arequest for metadata related to the content being consumed.Specifically, the request for metadata may include the audio profilegenerated in step 703. Thus, the audio profile may be transmitted fromthe second screen device 302 to a server (e.g., the second screenexperience computing device 340) configured to evaluate the audioprofile to identify content based on the audio profile. At step 705, thesecond screen experience computing device 340 may identify the contentbeing streamed based on the audio profile captured in step 703. Further,in step 705, the second screen experience computing device 340 maydetermine whether metadata tags exist for the identified content. Insome cases, the second screen experience computing device 340 mayspecifically search for metadata tags associated with the segment of thecontent related to the audio profile. As also discussed above, theidentification of the program may be done without using audio samplingand/or generating an audio profile. For example, the user may simplyview a program schedule or program listing on the second screen device302, and select one to identify it to the computing device 340. If theuser used the second screen device 302 to initiate playback of thecontent in the first place (e.g., using a device 302 that is paired tocontrol a DVR or STB, or to request a VOD session), the second screendevice 302 may determine from this selection what content is beingpresented on the first screen device 301.

In some embodiments, the second screen experience computing device 340may respond to the transmission in step 704 by providing the metadatafor any existing metadata tags or by providing a message indicating thatone or more metadata tags exist, and thus, the determination in step 705may be performed by the second screen device 302 itself.

If metadata tags do not exist (No at step 705), the second screenexperience computing device 340 may send a message indicating that nometadata tags exist to the transmitting second screen device 302.However, if metadata tags do exist, the metadata of the correspondingmetadata tags may be received at step 706. The received metadata tagsmay be ranked in a particular order based on, for example, a userprofile associated with the receiving second screen device 302 and/orpriority levels of users who generated the metadata. In some cases, thereceived metadata may only include the metadata associated with theaudio profile sent to the server. In other cases, the received metadatamay be some portion or all of the metadata from a time period in thecontent starting with the audio profile sent to the server. Still inother cases, the audio profile may simply be used for identification ofthe content, and might not control what metadata is displayed. In anyevent, the received metadata may then be displayed at step 707. Asdiscussed above, in some embodiments, the user of the second screendevice 302 receiving the metadata may choose which metadata to display.Such selections may be used in ranking the metadata in subsequentaccesses of the data as described above. For example, a user's selectionof a particular metadata may increase that metadata's ranking such thatit appears higher in a list of available metadata when the same user oranother user subsequently receives the metadata.

After displaying the supplemental content, in some embodiments, the userof the second screen device 302 may rate the metadata at step 708. Forexample, the user may select whether the user “likes” or “dislikes” themetadata. Such selections may also be used in ranking the metadata insubsequent accesses of the data as described above.

At step 709, the process of FIG. 7 may determine whether the applicationis still running. If not, the process of FIG. 7 may end. However, if itis still running, the process may loop back to step 703 to processanother piece of audio. By intermittently (periodically) detecting audioprofiles at step 703, the synchronization of the metadata tags displayedon the second screen device 302 with the content provided on the firstscreen device 301 may be achieved. Further, steps of FIG. 7 may berepeated so that a user may view metadata for various content orsegments of content streamed on the first screen device 301. Notably,the content being streamed may change, and thus, the audio profiledetected in a subsequent implementation of step 703 may be for differentcontent than the previous implementation of step 703. In some cases, auser of the second screen device 302 may trigger the detection of audioprofiles at step 703. For example, when a user knows that he/she hasswitched channels, the user may instruct the application to detect a newaudio profile (i.e., proceed to step 703) so that he/she may view themetadata associated with the content currently provided on the firstscreen device 301.

FIG. 8 illustrates a diagram of an example screen 801 of an applicationconfigured to allow a user to view metadata on a display 802 of a secondscreen device 302. As shown in FIG. 8, the screen 801 may displaymetadata for a particular piece of content. In this case, the screen 801shows example metadata for a particular episode of the television showSeinfeld. A user of the application may choose the episode for whichhe/she wants to view metadata. Such a selection may be performed bydetecting an audio profile from audio of content streaming on a firstscreen device 301 as described above with respect to step 703.Alternatively, or additionally, the selection may be made by selectingfrom a list of available content that has existing metadata. Also, theapplication may have a search box 803 that allows users to search for aparticular piece of content.

Once content is chosen, the screen 801 may display the metadata for thatcontent. The metadata may be displayed in a variety of ways. Forexample, the metadata may be displayed in chronological order accordingto the time point or segment of the content that the metadatacorresponds to. In other words, referring to FIG. 8, if the event “Jerryis in line for soup” occurs within the Seinfeld episode before the event“Elaine sings,” then the metadata generated by username1 and username2for the “Jerry is in line for soup” event may be displayed first. Asshown in FIG. 8, there may be more than one piece of metadata contentfor each event. In such cases, the metadata may be ordered in variousways as well. For example, the metadata may be ordered based onrelevance so that more relevant pieces of metadata appear above lessrelevant pieces. Here, relevance may be determined manually by, forexample, an administrator, or automatically based on the viewing user'sprofile or based on a priority level of the users that submitted thedata. As shown in FIG. 8, the author/creator of the metadata may beidentified by, for example, a username (e.g., username 1, username 2,etc.). Each username may be assigned a priority level, and therefore,metadata created by a person having a username with a higher prioritymay appear above metadata created by a person having a username with alower priority. Referring to FIG. 8, username 1 may have a higherpriority level than username 2, and as such, the metadata of username 1may appear above the metadata of username 2. As explained herein, thepriority levels for usernames may be established based on variousfactors, including how frequently a username enters metadata or whetherothers approve of other metadata entered by a username. In someembodiments, higher priority levels may be given to preferred usernames,such as those associated with companies that wish to have their metadataappear higher in the list. Also, while FIG. 8 shows that metadata may bedisplayed in chronological order, in other examples, the metadata may bedisplayed on the basis of priority level alone. Still in other cases, asexplained herein, users may control which metadata is displayed usingfilters and/or by providing a profile for the system to utilize indetermining which metadata a user might be interested in. Further, thedisplay may be modified based on a search of the metadata using, forexample, the search box 803. The application may also be configured toallow a user of the second screen device 302 to re-order the metadatatags. For example, the user may click and drag the metadata submitted byusername3 related to Kramer's t-shirt to the top of the list becauseKramer is wearing that t-shirt at an earlier time in the episode.

FIG. 8 also illustrates that the application may allow users to commenton (or reply to) metadata created by others. For example, username4 maycomment on metadata generated by username3 to provide another placewhere an item may be purchased. The application may permit users torespond to other users' metadata in this manner by allowing them toselect on the metadata they wish to respond to and prompting them with ametadata entry form similar to the form shown in FIG. 4G. Yet anotherfeature of the application may be that users can rate metadata generatedby other users. For example, a user having username1 may select a “like”or “agree” icon 804 appearing next to metadata created by username8.Meanwhile, another user having username2 may select a “dislike” or“disagree” icon 805 appearing next to metadata created by username8. Insome examples, the results of feedback from other users (e.g., how manypeople agree and/or how many people disagree) may be displayed on thescreen 801. Further, in some embodiments, when a predetermined number of“dislikes” or “disagrees” are submitted in response to a single piece ofmetadata, that metadata may be removed or otherwise hidden. Therefore,if inappropriate metadata is submitted (e.g., X-rated content or X-ratedlanguage), other users may prevent that metadata from being displayed.As a result, “the crowd” may police the metadata content displayed bythe application. This result is illustrated in FIG. 8 which shows thatmetadata submitted by username8 is hidden so that it can only beaccessed via a link, and which shows that metadata submitted byusername9 is removed. Where the metadata is hidden such that it is onlyaccessible via a link, whether the link can be followed may depend on auser's profile, which may indicate a user's age.

The crowd may also be leveraged to improve the quality of the metadata.For example, where metadata is spoiled (e.g., includes a web link thatis no longer available, inaccurate, irrelevant, or otherwiseundesirable), the application displaying the metadata may be configuredto allow the crowd to remove the metadata. In some examples, to preventany one person from having too much control over the metadata, a certainnumber of removal requests may be required before the metadata isactually removed. Additionally, or alternatively, to address concernsabout spoiled web links a web crawler may be utilized to confirm whetherweb links still exist. Moreover, the web crawler may also evaluate theweb links to determine whether the web links still show relatedmetadata. Furthermore, when metadata is determined to be spoiled, thesystem (e.g., the second screen experience computing device 340) maynotify the creator of the spoiled metadata tag. Given this notice, thecreator may choose to re-tag the metadata or edit the metadata tag(e.g., provide a new web link).

Although the above description explains that the crowd may be leveragedto police the quality of metadata, it is also contemplated that one ormore administrator(s) of the system may have the authority to review themetadata tags as well. For example, an administrator may removeduplicates, delete inappropriate metadata, and update metadata if it isspoiled.

FIG. 8 further illustrates that the application may allow users toinitiate the generation of metadata tags from the screen 801 showingalready submitted metadata tags. Specifically, the user may select anyof options 806, 807, and 808. In different embodiments, differentoptions 806, 807, and 808 may be available. By selecting option 806, auser may initiate identification of the content, for example, byprocessing audio of the content to detect an audio profile which can beused for comparison with audio files or audio profiles of known content.This may then result in a time point of the content being tagged, andallow the user to enter metadata for the tag in a manner similar to thatshown in FIG. 4F. On the other hand, when a time point or segment of thecontent is already identified (e.g., in a case where audio iscontinuously processed), a user may select option 807 thereby triggeringa metadata entry form (like the one shown in FIG. 4G) to be displayed.The user may then complete the metadata entry form and submit it. Oncesubmitted, the metadata tag may then appear in the listing shown on FIG.8 so that the user can see his/her submission. Additionally, oralternatively, option 808 may include a text box in which a user mayenter text. When the user begins to enter text and/or submits the text,the application may capture a time point for the content so that thetext can be added into the listing at the appropriate place inchronological order. Another option (not illustrated) for enteringmetadata while viewing the listing of metadata tags shown on the screen801 of FIG. 8 may include allowing the user to click a location on thescreen 801 into which he/she wishes to add metadata. In other words, theapplication may generate a dialog box similar to the dialog box 403 ofFIG. 4H when a user clicks on a portion of the screen 801 and receivethe metadata through the dialog box. Once the metadata is submittedthrough the dialog box, the application may then insert the metadata taginto a position of the listing where the user clicked. In this manner,the metadata entered in the dialog box may be associated with a timepoint of the content based on its relationship (e.g., relative order) toother metadata tags in the listing that are already associated with timepoints of the content.

It should be understood that the screen 801 of FIG. 8 is an exampleintended to aid understanding of the present disclosure. In someembodiments, the screen 801 might not show the events (e.g., “Jerry isin line for soup,” “Kramer barges into apartment,” etc.), and instead,simply show the metadata in chronological order. Where the events areshown, they may be defined by the content creators (e.g., the producersof Seinfeld) or some other administrator(s) who has undertaken the taskof identifying events throughout the content. Also, rather thanincluding text to describe events, the events may be represented byimages, audio, video, etc.

Further, the application may be synchronized with content streaming on afirst screen device 301 so that the metadata displayed on the screen 801may correspond to the particular segments or time points of the contentstreaming on the first screen device 301. For example, when the portionof the Seinfeld episode where Jerry waits in line for soup is streamedon the first screen device 301, only the corresponding metadatagenerated by username1 and username2 may be shown. Alternatively,additional metadata for other segments of the content (e.g., metadatacorresponding to when Elaine sings) may be shown, but may appear belowthe corresponding metadata generated by username1 and username2.Further, a feature of the application may be to display differentmetadata as the streaming content changes, add metadata as the streamingcontent changes, move an icon (e.g., arrow 809) to point to differentmetadata as the streaming content changes, highlight different metadataas the streaming content changes, or to automatically scroll down a listof metadata as the streaming content changes. To accomplish this task,the application may process audio of the content streaming on the firstscreen device 301 to identify a starting point and may assume that thestreaming content continues to stream. Using such a starting point, theapplication may determine an approximate time when the user is viewingcertain events, and thus, may display corresponding metadata. In someembodiments, the application may continuously or periodically processaudio from the content streaming on the first screen device 301 tosynchronize the metadata with the streaming content.

FIG. 9 illustrates another embodiment for displaying metadata on ascreen 901 of a second screen device 302. As shown in FIG. 9, a leftframe 902 of the screen 901 may display a listing of content (e.g.,television shows, movies, videos, etc.). The listing of content may berecently watched content, favorite content of a user of the secondscreen device 302, or all for which metadata exists. In some cases, thecontent listing may include statistics beside each content named in thelisting showing how many metadata tags exist for that content.Additionally, or alternatively, the left frame 902 may includestatistics indicating which friends, family members, or other specifiedindividuals have commented on which content. Thus, a user can identifyand navigate to metadata generated by specific individuals. The leftframe 902 may also indicate which content is being streamed on a near-byfirst screen device 301 and/or for which piece of content metadata iscurrently being displayed in a right frame 903 of the screen 901. Forexample, the television show “Friends” in the left frame 902 of FIG. 9is highlighted indicating that metadata for “Friends” is being shown inthe right frame.

Meanwhile, FIG. 9 also shows an example of metadata that might be shownin the right frame 903. The metadata shown in the right frame 903 maychange as the content provided on the first screen device 301 is playedback. Specifically, the metadata shown in the right frame 903 may besynchronized with a time point or segment of the content on the firstscreen device 301 so that when a particular time point or segment of thecontent on the first screen device 301 is played back, the correspondingmetadata may pop-up in the right frame 903 of the screen 901. Tofacilitate synchronization, the second screen device 302 may detect theparticular time point or segment by continuously recording audio fromthe content on the first screen device 301 and analyzing the recording(e.g., detecting an audio profile). Using audio recognition processes,the second screen device 302 may determine a point in time of thecontent. Then, the second screen device 302 may compare the determinedpoint in time to time points associated with metadata for the content todetermine when and what metadata to display.

Alternatively, instead of continuously recording audio or continuouslyprocessing audio from the content to detect audio profiles, the secondscreen device 302 may intermittently (e.g., periodically) detect anaudio profile and use the audio profile to identify a time within thecontent as a reference point. Then, the second screen device 302 may usea clock to keep track of time in relation to the reference point. Forexample, from an audio profile detected by the second screen device 302,the second screen device 302 may determine that the content on the firstscreen device 301 is five minutes into the content. The second screendevice 302 may then use a clock to keep track of time from that point onso that the second screen device 302 may determine when the user is 7minutes and 33 seconds into the content at which point it may displaycorresponding metadata in the right frame 903 of the screen 901. In somecases, the screen 901 may include a clock 904 showing a running time ofthe content. This clock may allow the user to assist in synchronizingthe content. Specifically, if the user suspects that the time of theclock 904 does not match the time of the content, then the user maychoose to detect a new audio profile, for example, by selecting a button(e.g., the identify content option).

As explained above, FIG. 9 shows that different metadata may pop-up onthe screen 901 in synchronization with content provided on a firstscreen device 301. In some cases, a particular piece of metadata may bedisplayed on the screen 901 until the next metadata tag is reached. Ifthere are many metadata tags, the second screen device 302 may regulatehow frequently the metadata changes so that the metadata can be viewed.Also, in some cases, advertisements or other information may be shown inbetween different metadata tags.

Further, while FIG. 9 illustrates metadata corresponding to thetelevision show “Friends,” it should be understood that metadataassociated with a commercial being displayed on the first screen device301 may also pop-up on the second screen device. For example, when acommercial for a car is provided on the first screen device 301,metadata associated with the commercial (e.g., a website of a localdealer selling the car, a video of the car, etc.) may pop-up onto thescreen 901 of the second screen device 302.

It should be understood from the description herein that synchronizationmay be more readily accomplished because timestamps may be determinedwhen creating the metadata tag submissions. Further, even though not allmetadata in any one embodiment may be synchronized, synchronization ofeven a small portion of the metadata may be desirable.

Additionally, another aspect of the disclosure is to provide a methodand system for content creators, content providers, and/or advertisersto view what others (e.g., the “crowd”) are saying about their content,services, and/or products. Such advertisers, for example, may bepermitted to add and/or present information related to their product orservices (subject to any filters). As described above, users may controltheir second screen devices 302 to filter out metadata so that they mayonly view metadata they are interested in. Similarly, content creators,content providers, advertisers, etc. may be able to use devices (e.g.,second screen devices 302) to filter metadata created by the crowd toview metadata they are interested in. For example, a store may filtermetadata associated with one or more pieces of content to view metadatathat mentions its name or includes a web link to its website.

Although example embodiments are described above, the various featuresand steps may be combined, divided, omitted, and/or augmented in anydesired manner, depending on the specific secure process desired. Forexample, FIG. 7 may be modified so that a user inputs informationidentifying content for which the user wishes to view metadata insteadof detecting an audio profile in step 703 and transmitting the audioprofile in step 704. This patent should not be limited to the exampleembodiments described, but rather should have its scope determined bythe claims that follow.

We claim:
 1. A method, comprising: receiving, from a user device,supplemental data associated with primary content, wherein saidsupplemental data is related to a particular portion of the primarycontent by associating said supplemental data with audio data from saidprimary content; and providing data tags to associate said supplementaldata to particular portions of said primary content.
 2. The method ofclaim 1, wherein the user device is configured to allow a user to entersaid supplemental data and to detect the audio data from audio, which isoutput through another user device configured to present the primarycontent.
 3. The method of claim 1, wherein the audio data includes anaudio profile, representing audio of the primary content, detected at atime to which the supplemental data applies.
 4. The method of claim 1,wherein the audio data includes an audio sampling of the primary contenttaken at a time to which the supplemental data applies.
 5. The method ofclaim 1, further comprising identifying the primary content to which thesupplemental data applies based on the audio data.
 6. The method ofclaim 1, further comprising filtering the supplemental data based onidentity of the users submitting the supplemental data.
 7. The method ofclaim 1, further comprising filtering the supplemental data based oncomparisons of textual comment data included in the supplemental data.8. The method of claim 1, further comprising ranking the data tags basedon identity of the users submitting the corresponding supplemental data.9. The method of claim 1, further comprising storing genre informationspecifying a genre for each of the data tags.
 10. A method, comprising:detecting, by a second device, an audio profile from an audio portion ofcontent being presented on a first device; receiving, by the seconddevice, a user input including secondary information related to thecontent; correlating the user input with the audio profile to associatethe secondary information with a particular portion of the content; andgenerating a message to represent said association.
 11. The method ofclaim 10, wherein the message includes metadata of the content and thedetected audio profile.
 12. The method of claim 10, further comprising:identifying the content based on the detected audio profile to generatecontent identification information; and determining a time point in thecontent at which the user input is received, wherein the messageincludes the content identification information, the time point, and thesecondary information.
 13. The method of claim 12, wherein theidentifying of the content comprises: transmitting the detected audioprofile to a third device; and receiving the content identificationinformation from the third device in response to transmitting thedetected audio profile.
 14. The method of claim 10, further comprising:identifying the content based on the detected audio profile to generatecontent identification information; and determining a real time point atwhich the user input is received, wherein the message includes thecontent identification information, the real time point, and thesecondary information.
 15. The method of claim 10, further comprisingreceiving a response indicating whether the message was accepted orrejected.
 16. A method, comprising: providing content to a first userdevice; receiving, from a second user device, a message comprising audioprofile data and supplemental data, wherein said audio profile datacorrelates said supplemental data to a particular portion of saidcontent; identifying the content based on the audio profile data; andassociating said supplemental data with the particular portion of saidcontent.
 17. The method of claim 16, further comprising: receiving, fromthe second user device, a metadata request comprising an audio profilerepresenting audio of the content at a specific time point within thecontent; retrieving one or more metadata tags based on the audio profileof the metadata request; and transmitting the one or more retrievedmetadata tags to the second user device, wherein the one or moretransmitted metadata tags include metadata for the specific time pointwithin the content.
 18. The method of claim 17, wherein the one or moretransmitted metadata tags are ordered based on a user profile associatedwith the second user device.
 19. The method of claim 17, wherein the oneor more transmitted metadata tags are ordered based on priority levelsof users who provided the metadata.
 20. The method of claim 17, whereinthe one or more transmitted metadata tags are transmitted insynchronization with the content playing on the first user device.